perm filename ISI.CL[COM,LSP] blob
sn#819595 filedate 1986-06-22 generic text, type C, neo UTF8
COMMENT ā VALID 00002 PAGES
C REC PAGE DESCRIPTION
C00001 00001
C00002 00002 ā20-Jun-86 1243 berman@isi-vaxa.ARPA CL support
C00056 ENDMK
Cā;
ā20-Jun-86 1243 berman@isi-vaxa.ARPA CL support
Received: from ISI-VAXA.ARPA by SU-AI.ARPA with TCP; 20 Jun 86 12:41:59 PDT
Received: by isi-vaxa.ARPA (4.12/4.7)
id AA16157; Fri, 20 Jun 86 12:41:00 pdt
From: berman@isi-vaxa.ARPA (Richard Berman)
Message-Id: <8606201941.AA16157@isi-vaxa.ARPA>
Date: 20 Jun 1986 1240-PDT (Friday)
To: RPG@SU-AI.ARPA
Cc: berman@isi-vaxa.ARPA
Subject: CL support
Here's ISI's proposal showing our plans to support the CL community:
COMMON LISP SUPPORT
1. Background
The effort by the Common Lisp community to define and propagate a Common Lisp
Standard is well under way. At the just concluded Common Lisp Meeting,
agreement was reached to pursue such standardization under ISO. Bob Mathis,
who helped guide the ADA standardization and validation efforts from his post
as head of AJPO, has been chosen as the convener of an expanded technical
committee to address the remaining technical issues in defining the Common Lisp
Standard and to resolve technical questions relating to adopted portions of the
standard.
However, the benefits arising from such standardization will not be realized
unless DARPA builds the infrastructure needed to nurture and support the
emerging Common Lisp community. The Common Lisp specification must be
correctly maintained and distributed to those who have need of it. A
validation suite is needed to guide developers toward proper implementations
and to ensure that they have correctly done so. A library of public domain
information concerning Common Lisp (validation suites, useful utility and
functional programs, documentation, tests, implementation guides, etc.) must be
maintained and disseminated. Past and future proceedings of the Executive
Committee must be archived and organized for retrieval by date and topic.
Libraries of public domain information concerning Common Lisp (source and
object code, documentation, tests, implementation guides, etc.) must be
maintained and disseminated. Network mailing lists of people and organizations
participating in the electronic forums used to raise issues, submit
suggestions, and arrive at consensus must be maintained. The messages must be
organized and summarized so that new people can join the forum and participate
in the distributed standardization effort. Finally, travel and administrative
support is needed for the convener of the Common Lisp standardization effort.
The Common Lisp community needs a technically competent support organization
to provide these services. Furthermore, this support group must have no stake
in any Common Lisp implementation so that they can perform the necessary work
with complete impartiality. ISI recognizes the need for these services and has
the technical and administrative expertise to support the Common Lisp
community. Moreover, ISI has the trust of that community. ISI is uniquely
qualified to support this effort because of its personnel, long history of
service support of the Lisp community, and demonstrated ability to manage and
operate service efforts. Bob Mathis was chosen as the convener by the Common
Lisp community. Ron Ohlander was chosen to be a member of the Common Lisp
Executive Board. Richard Berman has formed working relationships with the
validation experts in the vendor organizations. All of these people are ISI
employees. ISI has established good relations with many of these vendors
through its negotiation, purchase, and distribution of Lisp machines for
DARPA's Strategic Computing program. Prior to that, ISI produced, at DARPA's
request, a full compatible Interlisp for the Vax to promote its widespread use.
Several hundred copies have been distributed and the system has been
transferred to DEC for further distribution and maintenance. ISI has
successfully operated and managed the MOSIS service, a TOPS-20 resource center,
and remote support for DARPA's computing services. This combination of
required capabilities makes ISI uniquely qualified for this task.
2. Tasks
2.1. Common Lisp Validation
A public domain validation suite is desperately needed. Several purported
implementations of Common Lisp exist or will exist within the next few months.
The community has no way of determining the degree of compliance obtained by
such implementations. A few companies have extensive (though hardly
comprehensive) validation suites, but all these are proprietary. Building a
public domain validation suite would be quite expensive, would require
expertise in many different areas, and would take any one organization quite a
while to produce (because of staffing and phasing issues).
ISI proposes to pursue a very different approach. At the just concluded Common
Lisp Meeting, we chaired a discussion of the problems concerned with creating
such a public domain validation suite and investigated the alternative of
creating a public domain validation suite from vendor contributions. We found
much support for this alternative among the vendors if the resulting validation
suite would be public domain and if it was collected and maintained by a
technically competent, neutral, and non-commercial organization such as ISI.
This support was quite widespread and included both the vendors with existing
extensive proprietary validation suites and those who had not yet created such
proprietary suites. The former agreed to contribute their existing suite, and
the latter agreed to produce and contribute one for some portion of the Common
Lisp Standard. While these "agreements" are merely expressions of intent at
this point, we feel that they are based on a true sense of the needs of the
emerging Common Lisp community and the benefits that can accrue to the members
of that community. We have been working with these vendors to solidify these
"agreements" into hard commitments and have just begun to receive contributions
from these vendors. For information purposes, we have attached the list of
vendors that have informally agreed to contribute to a public domain Common
Lisp Validation Suite (see Attachment A).
ISI's role would differ for the two types of contributions. For the existing
suites, ISI must:
1. Homogenize the tests so that they are included in a common way and
report their results in a common way (currently, each vendor has
their own conventions).
2. Eliminate duplicate tests.
3. Determine the areas of coverage and the degree of coverage within
those areas.
For the newly produced suites, ISI must:
1. Specify the way that tests will be invoked and report their results.
2. Coordinate the focus of the vendors to maximize coverage.
In addition, ISI must:
1. Build a Validation Manager to invoke and collect results of
individual tests.
2. Determine the validity of each contributed test.
a. Correct any incorrect ones,
b. Forward questionable ones to the Common Lisp Technical
Committee for resolution of ambiguity.
3. Catalog each contributed test, identify its contributor(s), and
place it in an appropriate area of the validation suite.
4. Build and maintain a Public Domain Validation Suite library and
provide access to it.
Finally, ISI must conduct the validation of vendor's implementation. This will
be done by ISI personnel visiting the vendor site. The vendor will provide
visiting validation team access to the software to be validated and the
hardware upon which it runs. The ISI validation team will merely run the
validation suite, collect its results, and submit them to the Common Lisp
Steering Committee. The Committee will evaluate the results and determine the
degree to which the vendor's implementation complies with the Common Lisp
Standard.
2.2. Library of Public Domain Common Lisp Information
ISI will collect and organize public domain Common Lisp information
(currently over 20MB), maintain it in a library, update it as new information
is generated or becomes available, and provide a dissemination mechanism for
it.
We will make use of the ISI, DARPA funded, Common Lisp Framework (CLF)
persistent object oriented data base to store, house, locate and retrieve the
information types below, to link it with other information and to provide
release and version management. A network interface to the CLF data base will
be constructed to permit online retrieval of the archived information. In
addition, CLF's active object base mechanisms will be used to construct an
automated dissemination facility (triggered by new versions of the stored
objects), and an automated postmaster which responds to stylized requests for
information that arrive via electronic mail.
The Network Services group in the Intelligent Systems Division of ISI will be
tasked to accomplish much of the library and document maintenance services.
This group currently exists and is chartered to provide expeditious, useful and
reliable administrative computer and computer network support and service to a
number of clients of the Institute as well as to funded ISI researchers and
support staff. This group has historically been able to bring to the community
a core staff of competent maintenance personnel who will be able to coordinate
and provide the expertise required to help act as a clearing house and a
communications link for the various library activities.
The specific information and procedures, which will be dealt with by a
combination of Network Services and research staff overseeing this portion of
the overall effort, will include the following:
1. Lists of Common Lisp users and implementers: It will be necessary to
keep current lists of all Common Lisp user sites and implementers in
order to ensure proper delivery of appropriate documents. ISI will
undertake this task, distributing the lists to parties that require
them.
2. The Common Lisp specification: The specification will constitute the
baseline document which at all times determines the Common Lisp
language. This baseline document will require strict configuration
management to determine that it is kept appropriately up to date,
while at the same time preserving the integrity of the language.
Proposed changes to this document will me developed, evaluated, and
approved by the Common Lisp Technical Committee. All approved
changes will made by ISI staff. To support this change process, ISI
will collect and coordinate all requests for changes. Such requests
will be consolidated and forwarded to the Technical Committee for
their consideration. ISI will collect information concerning the
actions of the Technical Committee and report results back to
interested implementers and users. Successful use of the
specification may require other explanatory documents for successful
implementation. ISI will develop and distribute such documents, as
well as the specification itself. In addition, ISI will answer
calls and queries concerning technical issues regarding
interpretation and implementation of the Common Lisp specification.
3. The validation tests collected under the previous task: When the
validation tests have been completed, they will be collected and
maintained in a coordinated and accessible database for later
perusal, modification, extension, and dissemination. There are a
number of different mechanisms available which will be applicable
for this chore including: shared Electronic Bulletin Boards,
extensive and moderated mailing lists to the community, and
non-electronic newsletters. The validation test suite will be made
available to whoever requests it so that they can evaluate
implementations of Common Lisp. Instructions for use for use of the
validation test suite will be maintained and provided to interested
parties. Liaison hotlines will be manned to answer queries and to
troubleshoot problems in proper application.
4. An on-line version of the Common Lisp Manual: Maintaining an updated
and current version of the Common Lisp Manual will best be
accomplished by the Documentation Specialist in the Network Services
group under the guidance of a language expert and the Technical
Committee. This document will be periodically changed to correct
errors, extend or refine explanation, and to reflect changes to the
Common Lisp specification. In addition, ISI will maintain a service
to answer phone calls and written queries requesting interpretation
of the manual and to document requests for needed changes. Change
requests will be consolidated and prospective changes that respond
to legitimate requests will be drafted for consideration by the
Technical Committee. Approved changes will be implemented and
revised manuals will be sent to the Common Lisp community.
5. Public domain implementations of Common Lisp or "Yellow Pages"
additions to it: ISI will collect all implementations of Common Lisp
compilers, interpreters, environments, and auxiliary programs
offered by the community for use by other interested sites.
Furthermore, ISI will publish a catalogue of these programs on a
shared Electronic Bulletin Board. Programs will be available
through electronic file transfer from ISI or will be shipped to
interested parties. ISI will put parties who have implementation or
utilization questions into contact with developers of the original
software.
6. Contributed implementation guidelines and hints: Implementation
guidelines and helpful hints contributed by the user community will
be maintained at ISI through a database and historical record of
previous Common Lisp implementations. These archives will help
vendors/contributors to easily access the information they need to
get themselves on the right track for producing their own properly
validated version of Common Lisp. An automated dissemination
facility and automated postmaster will take care of the bulk of the
queries and problems. Liaison hotline services and inhouse research
staff available will resolve the exceptions.
7. Electronic forums on Common Lisp issues: One of ISI's strong suits
is the maintenance and moderating of electronic forums on a variety
of issues. A Common Lisp bulletin board and/or a master mailing
list of principals and players will be effectively managed from ISI.
8. Proceedings of the Common Lisp Executive Committee: Minutes of
Common Lisp Executive Committee meetings will be collected,
archived, transcribed, and selectively published by ISI. This will
be accomplished through the established databases and tools of CLF
as outlined above.
Since there will be a variety of requirements for distribution of requests for
information, for test validation suites, and for other information, ISI is an
ideal site to center this general distribution activity. Hopefully, most of
the distributions will occur electronically via the Internet, DDN and various
other Networks. When this is not possible due to the electronic isolation of
some vendor or site, ISI will be able to transmit information in nearly any
format required including: Magnetic Tape (nearly any format or density), floppy
disk and paper/hardcopy. ISI has a large mix of computers and peripherals as
well as a qualified and experienced Operations and Systems staff which will
allow for this variety of media production (as required). Additionally, with
an Operations staff available 24 hours-a-day, 7 days-a- week, researchers can
be assured maximum availability to the machine resources they might require or
expect to find.
2.3. Travel and Administrative Support for Convener
Bob Mathis, an ISI employee, was chosen as the convener of the Common Lisp
Standardization effort by the community. In this role, he will need substantial
administrative support and will be required to travel extensively, both
domestically and internationally, to attend the various standardization
meetings being held and to coordinate this standardization effort with
appropriate technical organizations. We estimate that 50 to 80 days of his
time and extensive travel, especially internationally, will be required each
year.
3. Current Activity and Plans
3.1. Validation Suites
ISI has established communication with the various Common Lisp vendors
regarding their previously "agreed upon" contributions to the public domain
validation suite and set in motion the process of collecting the existing
validation suites. Most of the vendors are now in the process of sanitizing
some or all of their existing validation suites to remove proprietary and/or
non-portable material prior to delivering the source code to ISI. Only a small
fraction of the expected contributions have actually been received.
As the source code trickles (and then, hopefully, pours) in, it will be
necessary to "re-sanitize" them into some common form. Nearly all of the
vendors are using some form of test management software developed along with
their tests. As one could expect, the nature of this test management is quite
different from vendor to vendor. Many are using some form of proprietary error
control mechanism, an area as yet not standardized in Common Lisp.
From the technical (rather than administrative) point of view, a number of
tasks must occur in order to successfully achieve the desired result of a truly
portable "universal" test suite for Common Lisp. The primary task is one of
understanding. Anyone who has ever had to maintain or extend source code
created by another knows the inherent communication problem in reaching a full
understanding of the source code. In this particular case, we have added the
complexity of trying to understand source code which has not only been written
by a number of others, but is written in and with the purpose of testing an
evolving language. As anyone in the Common Lisp development community could
attest, there are questions as to the exact meaning of a number of areas in the
current language specification (such as it is).
Thus the task becomes one of heavy liaison between not only the ISI technical
personnel responsible for the creation and maintenance of the validation suite
and the vendors, but also between the validation personnel and the technical
steering committee and the general implementors and development community. It
would not be at all surprising if, due to questions raised during the building
of the validation suite, there arose (and subsequently were resolved) a number
of language design and implementation issues.
Once each individual test (of which there may be hundreds or thousands per
vendor contribution) has been understood and validated as to being a test of
actual language features, and also of being a valid test of those features, it
must be incorporated into some kind of framework under which all of the
validation testing occurs. A few vendors have offered to contribute their own
validation managers, but as of this time these have not been received for
evaluation.
The most probable form of organization for the test suite is by sections
numbered as in Steele's book, Common Lisp, the Language. Thus, an individual
test must be exactly "locatable" as to the section in the book which discusses
and explains the feature being tested. Because this book has been much
clarified and revised by discussion over the network, and, in fact, because
this process is ongoing, there is required the development of some "meshing"
between the validation effort and the continuing specification effort.
Naturally, for any such validation suite to be effective it must be both
broad in terms of language issues covered, and deep in the extent of exercising
the full range of functionality of each language feature. The ideal validation
would exhaustively attempt every type of operation with every type of operand,
both legal and intentionally illegal. It is just as important to test that
error conditions are detected and reported correctly as it is to test for
simply correct functionality of legal expressions.
One of the initial hurdles that must be overcome in the creation of a
validation test management program is the nature of the detection and reporting
of exception or error conditions. The common consensus is to use "the error
handling mechanism" of the language. Indeed, all of the vendors who have
offered to contribute test management software, use the language error handling
features. Unfortunately at present there is no standard for the exact
definition and handling of exception conditions, and so each vendor has
implemented their own form of error handling. Either a portable
non-exception-based test control mechanism must be devised, or the error
handling features must be put in place in the language specification so that it
can be relied upon during validation.
During this initial phase of soliciting code contributions from vendors'
existing validation suites, ISI will also be evaluating those contributions
against the idealized validation suite described above. As more vendors
specify the exact nature of their test contributions, ISI can use this survey
to identify the holes in the composite suite formed from the contributions.
Several vendors have indicated that they would be willing to create wholly new
validation tests for specific areas of the Common Lisp Manual. We will
coordinate these vendor activities to maximize coverage and eliminate
duplication. Once a validation framework is decided upon for the individual
tests, we can then specify the format for all these new tests so that the
effort of integrating them into the full validation suite is minimal.
Besides these mostly technical tasks, there is a large administrative
responsibility in the ongoing communication with validation contributors, the
technical committees, the language developers and the Common Lisp user
communities. Not all developers have network access in a form that makes
distribution over the network possible. In the interest of full support for
all vendors and developers, the public domain validation suite should be made
available on tape and in other ways that would facilitate its distribution.
When revisions and/or additions are made to the validation suite, the
interested parties must be notified. Undoubtably, when a validation suite is
made public domain, there will not be complete agreement as to the validity of
the test itself. This could be despite approval by the various committees
involved. Especially in its initial incarnations there will certainly be a
succession of incremental revisions designed to correct flaws in the individual
tests of the validation suite.
To maintain complete impartiality, and to provide for a uniform standard of
reporting, when a vendor desires an official validation rating ISI will conduct
the test of the implementation. By going to the vendor's site we avoid the
need to send hardware to ISI, and all the subsequent problems that revolve
around that. A validation team would actually use the validation suite to
verify the implementation's correctness. The results of this official
validation run would be collected by the validation team, put into a formal
report in a standard format, and forwarded to the Technical Committee for
evaluation. By using this more formal approach, it will be possible for the
Technical Committee to objectively compare different implementations for
determining the level of compliance with the full Common Lisp standard.
3.2. Keeping Track of Decision Making and the "Why's"
Another area of ISI's participation as an administrative body for the Common
Lisp community is technical record keeping. Currently, a great deal of
discussion occurs on the network regarding specific details of the language,
its specification and implementation. Often an apparent consensus is reached
to alter or drop "old" language features, or to add entirely new ones. Yet
there does not appear to be any official statement from a technical steering
committee regarding any decisions to accept proposals. And too, often there is
no formal proposal, but simply a semblance of agreement amongst those arguing a
point.
It is not unusual to see something like:
- As I recall, we discussed point "x" several months ago, but I can't
seem to recall what we decided...
Clearly, this causes repetition of whole discussions, and in general throws
rocks in what is already an unpaved road. Ideally there should be some method
of classifying both the content and nature of each message, as well as some
form of discussion tracing, all built into a kind of archiving methodology
which provides querying and possibly browsing facilities. Once such a
record-keeping system is in place, further discussion participants can be
advised as to the classification system so that they can indicate the
classification for their messages. In the meantime, and especially while
developing the classification method, each message will have to be classified
at ISI.
During these discussions, many valid points are raised and, often, resolved.
A great deal of the philosophy of the language design is contained in these
messages. There are a number of ways in which these messages can be classified
and organized. There are two main areas of classification:
1. The "type" of message.
2. The topic(s) of the message.
While it would be better if a message contained exactly one topic, in
practice even one "topic" easily branches off into several related subjects
which wouldn't be appropriate to separate messages. Thus each message would be
classified by a list of topics discussed. Often this would be a specific
Common Lisp function or feature. At other times it might discuss a more
general area, such as error handling, or string representation. It may be
feasible to assume that there is a finite set of topics into which the more
specific topics can fit. For example, "string representation" might fit under
"strings", as would discussions about specific string functions.
When classifying by topic (and especially when considering a finite set of
topics) there is a useful guide -- Steele's book, Common Lisp, the Language.
The list of finite topics could well be the section numbers in that book, so
that when discussing a particular topic or topics, the message sender must know
what sections of the book contains the material which he has some discussion
concerning. This has the added advantage of making the message sender more
aware of the published information regarding his interest. Thus, if this were
the decided-upon classification method, each message on the network concerning
Common Lisp would have a list of section numbers, put into the message by the
sender.
The querying or browsing facility mentioned above would then allow the user
to view (and summarize, count, etc.) the messages regarding a certain topic.
Naturally these messages would appear in temporal sequence. A typical request
might be to summarize (i.e. print the sender, date and discussion topic) of
each message on a given set of topics within a certain time period. This would
help to locate a specific message by jogging the searchers memory.
The other main form of classification is by "type" of messages. A key type
might be "Ratified Decision", or some similar type. This would be a message
(from the steering committee) officially announcing some change to the language
specification. Like any other message, it would be classified by topic(s). By
using a type-classification as well, a user could now search for "decisions
made about strings between February and June of 1986". If one only cared for
specification changes, this would be an efficient means of getting the concise
information needed.
Other types can be suggested, such as "Start of new discussion", which would
indicate the first message in a new chain of discussion. Or "Clarification
request", "Proposal", etc. Undoubtably the process of building this
record-keeping mechanism will bring to light the required set of message types.
Another valuable means of organizing these messages is by discussion tracing.
By this is meant the linking, in temporal sequence, of each message in a given
discussion, from the inception of the discussion through to its resolution. In
a browsing mode, the user might be able to start from any message and trace the
discussion chain forward and backward to get the full set of things considered
when resolving the discussion. This is especially useful because a great deal
of traffic is currently generated in repeating discussions resolved earlier.
Probably other ways of organizing the Common Lisp discussions will present
themselves as the task progresses. By using the Common Lisp Framework
developed at ISI, each message can be treated as a unique object, and given
properties (such as type and topics) which relate the messages one to another.
The facilities provided under CLF allow for complex searching to be done
amongst these objects. A stylized querying language and a reporting method can
be created so that network users can access the CLF Common Lisp data base to do
the research they need to more optimally interact with the development
community.
4. Milestones
- 3 Months
* Build a Validation Manager to invoke and collect results of
individual tests.
* Coordinate the focus of vendors contributing to Validation Suite
to maximize coverage.
- 6 Months
* Integrate initial contributed Validation Suites into Library
1. Homogenize the tests so that they are included in a common
way and report their results in a common way (currently,
each vendor has their own conventions).
2. Eliminate duplicate tests.
3. Determine the areas of coverage and the degree of coverage
within those areas.
- 9 Months
* For each test in the initial contributed Validation Suites:
1. Determine validity of test
- Revise or remove incorrect tests
- Forward questionable ones to Technical Committee
2. Place test in appropriate portion of Validation Suite
3. Catalog it by its contributor, the test it performs, and
its time of contribution
- 12 Months
* Make information in the Common Lisp Library accessible. This
information includes:
1. The Common Lisp specification.
2. The Public Domain Validation Suite.
3. An on-line version of the Common Lisp Manual.
4. Public Domain Implementations of Common Lisp or "Yellow
Pages" additions to it.
5. Contributed implementation guidelines and hints.
6. Electronic forums on Common Lisp issues.
7. The Proceedings of the Common Lisp Executive Committee.
- 15 Months
* Design procedures by which on-site validation of vendor
implementation of Common Lisp will be carried out. Coordinate
with vendors and Common Lisp community.
- 18 Months
* Automated Postmaster and Dissemination mechanism for Common Lisp
information
* Begin conducting on-site validation of vendor implementations
5. Computing Resources
ISI does not require any additional (DARPA purchased) Common Lisp
Workstations for this project. The necessary workstations will be supplied
from a pool of grant Common Lisp Workstations given to ISI by Hewlett Packard
and Texas Instruments in recognition of DARPA sponsored work in the Software
Sciences Division on FSD.
Operations and maintenance costs of these workstations, however, must be
included in the budget for this effort.
-------